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Executive  Summary 


This  research  aims  to  develop  new  and  more  accurate  stochastic  models  for  speaker-independent 
continuous  speech  recognition  by  extending  previous  work  in  segment-based  modeling  and  by  intro¬ 
ducing  a  new  hieraLrchical  approach  to  representing  intra-utterance  statistical  dependencies.  These 
techniques,  which  have  high  computational  costs  because  of  the  large  search  space  associated  with 
higher  order  models,  are  made  feasible  through  rescoring  a  set  of  HMM-generated  N-best  sentence 
hypotheses.  We  expect  these  different  modeling  techniques  to  result  in  improved  recognition  per¬ 
formance  over  that  achieved  by  current  systems,  which  handle  only  frame-based  observations  and 
assume  that  these  observations  are  independent  given  an  underlying  state  sequence. 

In  the  past  quarter,  our  focus  has  been  on  developing  the  theory  and  initial  implementation 
behind  high  level  models  and  search  algorithms  to  accommodate  these  models.  These  eflfbrts  and 
other  accomplishments  of  this  project  include: 

•  further  investigation  of  different  variations  of  mixture  distributions,  looking  in  particular  at 
robust  estimation  and  initialization  issues; 

•  development  of  a  new  approach  to  continuous  density  parameter  adaptation; 

•  development  of  training  algorithms  for  a  discrete  distribution  intra-utterance  correlation 
model  and  two  methods  for  applying  the  model  in  multi-pass  recognition  scoring; 

•  conducted  further  experiments  with  the  sentence-level  mixture  language  model,  demonstrat¬ 
ing  performance  improvements  on  the  5k  vocabulary  WSJ  hub  test  set; 

•  implemented  much  of  the  software  needed  for  lattice  rescoring  with  the  SSM; 

•  modified  our  distribution  clustering  algorithm  to  allow  general  splits;  and 

•  began  exploring  auditory  signal  processing  algorithms  on  the  TIMIT  phone  recognition  task. 

We  also  continued  to  make  minor  improvements  to  our  baseline  recognition  system.  Our  current 
best  performance  figures  with  the  BU  acoustic  and  language  model  are  7.0%  and  5.7%  error  rates 
on  the  5k  WSJ  development  and  evaluation  test  sets,  respectively.  These  numbers  improve  to  6.3% 
and  5.3%  error  rates  when  we  also  include  the  BBN  HMM  and  SNN  acoustic  scores  in  reranking 
the  sentence  hypotheses. 
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2  Summary  of  Technical  Progress 

Introduction  and  Background 

In  this  work,  we  are  interested  in  the  problem  of  large  vocabulary,  speaker-independent  contin¬ 
uous  speech  recognition,  and  primarily  in  the  acoustic  modeling  component  of  this  problem.  In 
developing  acoustic  models  for  speech  recognition,  we  have  conflicting  goals.  On  one  hand,  the 
models  should  be  robust  to  inter-  and  intra-speaker  variability,  to  the  use  of  a  different  vocabulary 
in  recognition  than  in  training,  and  to  the  effects  of  moderately  noisy  environments.  In  order  to 
accomplish  this,  we  need  to  model  gross  features  and  global  trends.  On  the  other  hand,  the  models 
must  be  sensitive  and  detailed  enough  to  detect  fine  acoustic  differences  between  similar  words  in 
a  large  vocabulary  task.  To  answer  these  opposing  demands  requires  improvements  in  acoustic 
modeling  at  several  levels:  the  frame  level  (e.g.  signal  processing),  the  phoneme  level  (e.g.  model¬ 
ing  feature  dynamics),  and  the  utterance  level  (e.g.  defining  a  structural  context  for  representing 
the  intra-utterance  dependence  across  phonemes).  This  project  addresses  the  problem  of  acoustic 
modeling,  specifically  focusing  on  modeling  at  the  segment  level  and  above.  The  research  strategy 
includes  three  main  thrusts.  First,  phone-level  ^oustic  modeling  is  based  on  the  stochastic  seg¬ 
ment  model  (SSM)  [1,  2],  and  in  this  area  our  main  efforts  involve  developing  new  techniques  for 
robust  context  modeling,  mechanisms  for  effectively  incorporating  segmental  features,  and  models 
of  within-segment  dependence  of  frame-based  features.  Second,  high-level  models  are  being  ex¬ 
plored  in  order  to  capture  speaker-dependent  and  session-dependent  effects  within  the  context  of 
a  speaker-independent  model.  In  particular,  we  are  investigating  hierarchical  structures  for  rep¬ 
resenting  the  intra-utterance  dependency  of  phonetic  models,  and  more  recently  language  models 
for  representing  topic  dependency  and  language  dynamics,  recognizing  that  higher-order  models 
of  correlation  can  extend  to  the  language  domain  as  well  as  the  acoustic  domain.  Lastly,  speech 
recognition  is  implemented  under  the  N-best  rescoring  paradigm  [3],  in  which  the  BBN  Byblos  sys¬ 
tem  is  used  to  constrain  the  SSM  search  space  by  providing  the  top  N  sentence  hypotheses.  This 
paradigm  facilitates  research  on  high-order  models  through  reducing  development  costs,  and  pro¬ 
vides  a  modular  framework  for  technology  transfer  that  has  enabled  us  to  advance  state-of-the-art 
recognition  performance  through  collaboration  with  BBN. 
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Summary  of  Technical  Results 


In  the  first  half  of  this  project,  we  have  focused  on  improving  the  performsmce  of  the  basic  segment 
word  recognition  system  and  porting  the  system  to  the  Wall  Street  Journal  task  domain.  In  brief, 
the  accomplishments  of  that  period  included:  improvements  to  the  N-Best  rescoring  weight  esti¬ 
mation  algorithm;  investigation  of  different  mechanisms  for  improving  the  baseline  acoustic  model, 
including  distribution  clustering  [4],  mixture  modeling  at  different  time  scales  [5,  6],  theoretically 
consistent  models  based  on  context-dependent  posterior  distributions,  automatic  distribution  map¬ 
ping  estimation,  and  hierarchical  models  of  intra-utterance  phoneme  dependence;  implementation 
of  baseline  n-gram  and  sentence-level  mixture  language  models;  and  improvements  to  the  SSM 
baseline  system  aimed  at  improving  performance  on  the  ARPA  WSJ  task  and  participation  in  the 
benchmark  tests. 

The  research  efforts  during  this  quarter,  supported  in  part  by  an  ONR  AASERT  award,  have 
primarily  involved  theoretical  and  initial  software  development  for  models  of  high-order  correlation 
and  search  algorithms  to  accommodate  these  models.  These  efforts  and  other  research  developments 
are  summarized  below. 


Mixture  distributions  in  the  SSM.  We  have  been  attempting  to  improve  the  performance 
of  the  untied  mixture  system,  both  for  the  independent-frame  model  and  the  segmental  mixture 
model,  investigating  issues  related  to  robust  parameter  estimation.  One  iss'te  that  arises  in  training 
mixtures  is  that  care  must  be  taken  to  ensure  that  covariance  estimates  are  robust.  The  EM  algo¬ 
rithm  for  training  mixtures,  like  all  maximum  likelihood  (ML)  methods,  suffers  from  the  difficulty 
that  the  training  likelihood  can  be  increased  without  bound  if  a  Gaussian  mean  is  centered  on  one 
of  the  traning  observations  auid  its  covariance  is  made  to  approach  a  singularity.  ML  training  algo¬ 
rithms  accordingly  tend  to  naturally  converge  to  such  degenerate  solutions  if  not  corrected  in  some 
way.  We  investigated  a  number  of  variance  “clipping”  schemes  for  diagonal  covariance  Gaussians, 
patterned  after  the  approach  given  in  [7].  In  these  experiments,  the  degree  of  clipping  was  varied 
according  to  the  amount  of  training  for  the  Gaussian.  We  also  looked  at  clipping  the  variance 
based  on  a  percentage  of  two  different  values  -  either  a  histogram  of  the  estimated  variances  or 
an  estimate  of  the  overall  feature  variance.  Unfortunately,  these  methods  led  to  only  very  small 
improvements  over  the  previous  method  of  clipping  with  a  single  threshold,  regardless  of  training. 
Investigating  this  further,  we  found  that  other  researchers  have  observed  similar  results  [8].  We 
have  also  been  investigating  alternate  initialization  procedures,  including  one  that  starts  from  a 
single  Gaussian  per  mixture  and  successively  increases  the  number  of  components  by  “splitting” 
the  Gaussian  with  the  largest  likelihood.  This  approach  has  so  far  produced  inferior  performance 
to  our  previous  approach  of  initializing  from  non-mixture  SSM  models. 
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Adaptation  of  continuous  distributions.  It  is  well  known  that  speech  recognition  performance 
can  be  improved  by  matching  the  training  data  to  the  test  conditions,  whether  it  be  to  address 
channel/environment  effects  or  match  the  recognizer  to  a  particular  speaker.  In  most  cases,  however 
there  is  not  enough  training  data  available  to  handle  a  particular  speaker /environment  condition, 
and  automatic  adaptation  techniques  are  used  to  tune  a  general  modd  to  a  particular  condition.  In 
particular,  we  are  interested  in  unsupervised,  incremental  adaptation  of  Gaussian  density  param¬ 
eters,  which  is  more  difficult  than  adaptation  of  tied  mixture  parameters  (the  approach  currently 
used  by  most  sites)  because  of  the  large  number  of  parameters  to  adapt.  Our  approach  is  to 
combine  the  advantages  of  a  Bayesian  framework  [7,  9]  with  the  robust  properties  of  vector  fidd 
smoothing  [10]  to  adapt  a  large  number  of  continuous  densities.  To  handle  the  problem  of  adapting 
a  large  number  of  parameters,  we  define  a  two-levd  modd,  that  includes  a  large  set  of  distributions 
used  for  recognition  that  we  would  like  to  adapt  and  a  smaller  set  of  distributions  that  represent 
a  coarser  but  more  frequently  observed  space.  This  two  level  modd  is  built  using  divisive,  maxi¬ 
mum  likelihood  distribution  clustering,  which  gives  a  tree  with  detailed  models  at  its  leaves  (to  be 
adapted)  and  coarser  models  defined  by  a  pruned  version  of  the  tree  (for  which  adaptation  statistics 
are  computed).  Like  vector  field  smoothing,  we  use  a  wdght  function  for  determining  the  rdative 
contribution  of  adaptation  vectors;  but  in  our  case  the  wdght  is  based  on  a  measure  of  the  mutual 
information  between  the  coarse  and  fine  distributions^  rather  than  Euclidean  distance  as  used  in 
vector  field  smoothing.  An  advantage  of  this  framework  is  that  the  coarse  models  can  easily  be 
chosen  to  reflect  different  degrees  of  detail  and  thus  tuned  to  the  amount  of  adaptation  data  avail¬ 
able,  i.e.  a  small  set  of  adaptation  distributions  is  appropriate  for  a  small  set  of  adaptation  data. 
The  adaptation  algorithm  is  currently  being  implemented  for  uni-modal,  full-covariance  Gaussian 
mean  adaptation. 


Mixture  language  modeling.  One  of  the  important  questions  in  language  modeling  today  is 
how  to  effectively  represent  the  long-term  structure  of  language,  i.e.  how  to  capture  dependence  over 
longer  sequences  of  words  them  can  be  modeled  with  a  simple  n-gram.  To  address  this  problem,  we 
have  developed  a  sentence-level  mixture  language  model  (LM)  that  represents  the  topic-dependent 
structure  of  language  with  separate  n-gram  language  model  mixture  components  determined  using 
automatic  clustering.  We  have  continued  the  experimental  work  that  we  started  in  the  previous 
quarter,  but  in  our  recent  efforts  we  were  able  to  use  additional  language  model  training  data  ob¬ 
tained  from  BBN.  We  found  a  small  reduction  in  the  perplexity  due  to  the  mixture  language  models 
using  5  and  8  component  mixtures  to  model  the  5k  vocabulary  WSJ  task.  More  importantly,  we 
obtained  a  6%  reduction  in  the  recognition  error  using  the  5-component  mixture  models  as  com¬ 
pared  to  the  standard  trigram  models.  A  paper  reporting  these  results  was  presented  at  the  March 
1994  ARPA  Workshop  on  Human  Language  Technology  [11].  In  all  the  experiments  so  far  where 
we  sought  to  demonstrate  feasibility  of  the  model,  clustering  and  estimation  algorithm  approxima¬ 
tions  were  used  to  save  computation.  Current  efforts  are  aimed  at  implementing  more  theoretically 
motivated  algorithms,  such  as  a  similarity  measure  using  inverse  document  frequencies  and  full 
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Expectation- Maximization  (EM)  training,  which  we  expect  will  result  in  a  small  improvement  in 
the  modeb. 


Intra-utterance  phoneme  dependence  modeling.  We  further  developed  the  theoretical  frame¬ 
work  for  a  hierarchical  model  of  dependence  for  a  set  of  discrete  random  variables,  which  we  have 
begun  investigating  as  a  model  of  intra-utterance  phoneme  dependence.  We  use  a  dependence  tree 
[12]  to  represent  the  correlation  among  random  variables,  using  a  tree  structure  (designed  auto¬ 
matically)  with  Markov  assumptions  along  the  branches  of  the  tree.  The  dependence  tree  gives 
us  an  approximation  to  the  joint  distribution  of  the  set  of  random  variables  with  a  much  smaller 
number  of  parameters  than  a  full  joint  model.  We  can  apply  this  model  to  speech  recognition, 
by  defining  a  fixed-length  vector  (one  dimension  per  phoneme)  for  each  utterance  and  letting  each 
element  of  the  vector  take  on  a  discrete  value  (e.g.  vector  quantization  index)  based  on  all  of  the 
versions  of  the  phone  in  the  utterance  that  corresponds  to  that  element  of  the  vector.  Any  phone 
that  did  not  appear  in  the  utterance  is  considered  to  be  "unobserved”.  The  dependence  tree  model 
represents  a  distribution  for  this  vector  that  can  be  used  in  recognition  in  at  least  two  ways.  First, 
it  can  provide  a  "score”  of  the  consistency  of  the  phones  in  the  utterance  and  incorporated  as 
an  additional  knowledge  source  in  N-best  rescoring.  Second,  the  dependence  tree  can  serve  as  a 
prior  on  phone  distribution  parameters,  which  would  be  used  in  a  Bayesian  approach  pairameter 
adaptation  or  even  directly  in  recognition  scoring.  Since  we  have  unobserved  data,  the  Expectation- 
Maximization  (EM)  algorithm  is  required  for  pairameter  estimation.  Although  we  eventually  plam 
to  use  the  EM  adgorithm,  our  initial  implementation  of  the  training  algorithm  has  been  based  on  a 
simple  iterative  algorithm  that  substitutes  the  most  likely  observation  for  a  missing  component.  An 
initial  model  has  been  estimated  using  simple  cepstral  features  on  a  subset  of  the  TIMIT  corpus, 
which  gives  an  increase  in  traiining  data  likelihood  ais  expected.  Our  next  step  is  to  implement 
EM  training  and  build  a  model  on  the  full  corpus,  before  turning  to  the  implementation  of  the 
algorithm  for  recognition. 


Lattice  search  algorithms  for  multi-pass  recognition  scoring.  In  the  previous  quarter,  we 
developed  a  local  search  algorithm  for  lattice  rescoring,  motivated  by  the  good  performance  of  our 
recognition  system  on  the  WSJ  (which  suggest  that  we  could  improve  performance  by  considering 
more  hypotheses)  and  the  need  for  a  framework  that  can  accommodate  sentence-level  acoustic  and 
language  models.  We  plan  to  implement  both  optimal  and  local  lattice  search  algorithms,  so  as 
to  assess  the  performance/speed  trade-offs  experimentally,  and  in  this  quarter  we  initiated  soft¬ 
ware  development  for  extracting  the  lattice  and  dynamic  programming  search.  The  effort  involves 
both  developing  software  at  BU  for  performing  the  search  using  the  SSM  and  modifying  the  BBN 
decoder  so  that  it  produces  word  lattices.  So  far,  an  SSM  word  lattice  rescoring  algorithm  has 
been  implemented  that  handles  cross-word  triphone  models  and  trigram  language  models.  The 
algorithm  uses  hypothesized  phone  boundary  windows  to  reduce  computation,  where  the  windows 
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can  be  determined  from  phone  end  times  provided  by  the  previous  HMM  scoring  pass  or  by  a 
prediction  based  on  left-context  phone  duration  means.  Experiments  are  underway  to  verify  that 
performance  is  comparable  to  N-best  rescoring  and  to  assess  the  impact  of  using  predicted  phone 
end  times.  In  axldition,  we  modified  the  BBN  decoder  to  produce  lattices  from  the  N-best  traceback 
structure,  but  further  modifications  are  needed  to  produce  silence  information  (i.e.  the  number  of 
frames  of  silence  at  the  end  of  each  word)  and  optionally  phone  segmentations  for  each  word.  [This 
work  was  supported  by  the  ONR  AASERT  award.] 

Distribution  clustering  for  model  size  reduction.  In  previous  work,  we  developed  a  distri¬ 
bution  clustering  algorithm  to  specify  regions  of  parameter  tying,  effectively  reducing  model  size 
without  sacrificing  performance.  The  algorithm  was  based  on  divisive  clustering,  similar  to  that 
used  in  HMMs  for  state-dependent  clustering,  but  with  a  maximum  likelihood  criterion  specified 
according  to  the  specific  parameters  to  be  tied  [4].  One  of  the  limitations  of  any  tree-based  cluster¬ 
ing  algorithm  is  that  successive  divisions  of  the  training  data  can  lead  to  data  fragmentation  and 
less  effective  use  of  context  information.  This  is  potentially  a  greater  problem  for  our  system  in 
that  we  implemented  only  simple  questions  of  the  form  “is  the  tth  feature  in  set  A?”  To  partially 
overcome  this  problem,  we  implemented  complex  (multi-feature)  questions  using  a  version  of  the 
“pylon"  question  design  algorithm  [13].  A  pylon  contains  a  sequence  of  binary  questions  tied  so 
that  at  each  stage  there  are  only  two  subdivisions  of  the  data.  In  tree  design,  we  first  choose  a 
question  and  node  to  split  as  in  normal  growing  of  the  tree.  Then  a  pylon  growing  algorithm  is 
used,  which  successively  chooses  the  best  question  to  split  and  then  the  best  merge  out  of  two 
of  the  three  data  subsets,  where  both  splits  and  merges  are  evaluated  according  to  our  maximum 
likelihood  criterion.  This  procedure  is  continued  until  the  stopping  criterion  is  reached  (maximum 
number  of  questions  asked  in  the  pylon  or  no  question  further  splits  the  nodes)  and  the  remaining 
two  nodes  are  taken  as  the  child  nodes  produced  due  to  the  split  by  the  original  question  at  the 
splitting  node.  In  preliminary  results,  we  find  that  when  we  use  pylons  to  further  reduce  model  size 
by  about  50%,  we  observe  slightly  worse  performance  compared  to  the  normal  clustering.  More 
experiments  are  needed  to  determine  whether  performance  gains  are  achieved  when  model  size  is 
the  same  as  for  our  standard  clustering. 


Auditory-based  signal  processing.  We  have  begun  an  effort  to  assess  the  use  of  different 
signal  processing  algorithms  in  speech  recognition,  motivated  by  capabilities  of  the  human  auditory 
system.  Our  goal  is  to  demonstrate  improved  recognition  performance  for  dean  signals,  not  just 
signals  in  noise  as  is  typically  done  with  auditory  models).  To  reduce  the  scope  of  this  project, 
we  are  assessing  phone  recognition  performance  on  the  TIMIT  task  using  HTK  (Hidden  Markov 
Model  Toolkit).  With  the  help  of  Steve  Young  from  Cambridge  University,  we  have  duplicated  a 
system  based  on  cepstral  parameters  that  gives  one  of  the  best  reported  results  on  this  task  [14], 
and  we  are  currently  assessing  the  performance  of  the  Seneff  auditory  model  (available  through  the 
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FVontien  in  Sp«ech  Recognition  Workshop  CDROM)  using  a  similar  strategy  for  EMM  topdogy 
building.  Initial  results  are  not  encouraging,  but  we  hope  with  further  experiments  to  establish 
baseline  performance  at  least  close  to  that  for  cepstra,  and  then  to  consider  alternative  auditory 
models.  [This  work  was  supported  by  the  ONR  AASERT  award.] 

Future  Goals 

Based  on  the  results  of  the  past  year  and  our  ori^al  goals  for  the  project,  we  have  set  the  following 
goals  for  the  next  six  months:  (1)  continue  implementation  of  the  lattice  search  algorithm  and  as¬ 
sess  performance/speed  trade-offs;  (2)  further  develop  the  hierarchical  model  formalism  and  assess 
the  trade-offs  between  linear  and  non-linear  models  of  dependence;  (3)  extend  the  language  mod¬ 
eling  work  include  a  dynamic  component  and  to  handle  new  vocabulary  items;  and  (4)  investigate 
unsupervised  adaptation  in  the  WSJ  task  domain. 
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3  Publications  and  Presentations 

During  this  reporting  period,  we  published  one  conference  paper  (abstracts  are  reviewed),  and  gave 
an  additional  conference  presentation,  as  well  as  one  invited  talk  associated  with  this  project,  as 
itemized  below. 

“Language  Modeling  with  Sentence- Level  Mixtures,”  R.  Iyer,  M.  Ostendorf  and  J.  R.  Rohlicek, 
Proceedings  of  the  1994  ARP  A  Workshop  on  Human  Language  Technology,  March  1994. 

“Stochastic  Segment  Modeling  for  Continuous  Speech  ^cognition,”  M.  Ostendorf  et  al.,  presented 
at  the  March  1994  ARPA  Workshop  on  Spoken  Language  Technology. 

“A  Unified  View  of  Stochastic  Modeling  for  Speech  Recognition”,  M.  Ostendorf,  invited  talk  at 
ICSI,  Berkeley,  CA,  January  1994 

In  addition,  one  Boston  University  M.S.  thesis  proposal  was  successfully  defended  by  Fred  Richard¬ 
son,  entitled  “Lattice-Based  Search  Strategies  for  the  Stochastic  Segment  Model.” 
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4  Transitions  and  DoD  Interactions 

This  grant  includes  a  subcontract  to  BBN,  and  the  research  results  and  software  is  available  to 
them.  Thus  far,  we  have  collaborated  with  BBN  by  combining  the  Byblos  system  with  the  SSM 
in  N-Best  sentence  rescoring  to  obtain  improved  recognition  performance,  and  we  have  provided 
BBN  with  papers  and  technical  reports  to  facilitate  sharing  of  algorithmic  improvements.  On  their 
part,  BBN  has  been  very  helpful  to  us  in  our  WSJ  porting  efforts,  providing  us  with  WSJ  data  and 
consulting  on  format  changes.  We  have  also  begun  au  effort  to  collaborate  more  closely  in  lattice 
rescoring,  and  expect  that  Boston  University  student  Fred  Kichardson  will  implement  software 
libraries  that  wiU  be  shared  by  both  sites. 

The  recognition  system  that  has  been  devdoped  under  the  support  of  this  grant  and  of  a 
joint  NSF-ARPA  grant  (NSF  #  IRJ-8902124)  is  currently  being  used  for  automatically  obtaining 
good  quality  phonetic  alignments  for  a  corpus  of  radio  news  speech  under  development  at  Boston 
University.  The  alignment  effort  is  supported  by  the  Linguistic  Data  Consortium,  through  a  grant 
that  allowed  us  to  add  cross-word  phonological  rules  to  the  segmentation  software. 
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5  Software  and  Heirdware  Prototypes 


Our  research  has  required  the  development  and  refinement  of  software  systems  for  parameter  es¬ 
timation  and  recognition  search,  which  are  implemented  in  C  or  C-1-+  and  run  on  Sun  Sparc 
workstations.  No  commercialization  is  planned  at  this  time. 
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